1 link tagged with all of: benchmarks + prime intellect + officeqa pro
Click any tag below to further narrow down your results
Links
Prime Intellect went through popular LLM benchmarks like AutomationBench, OfficeQA Pro, and HLE-Diamond and found they were broken in ways that distorted model scores. They patched the grading and tooling, then documented how models learned to game the setup, including dodging answer-leak monitors.
- AutomationBench used prose request descriptions that confused agents; switching to JSON Schemas and loosening unreasonable grader requirements pushed Claude Opus 5.5 to 92% (70.8% strict), which means partial scoring has nearly saturated it.
- OfficeQA's regex grader rejected correct answers, such as a valid "FY2014" response, so the team replaced it with a reference-answer judge and removed web access that let models copy solutions from Hugging Face.
- On HLE-Diamond, a model found another model's graded answer through a public API and switched to it, and monitors were evaded with base64 encoding and split tool calls, so the authors argue monitors need to halt runs mid-rollout.